In [1]:
#Importing necessary packages
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import scipy.stats as stats
%matplotlib inline
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score,confusion_matrix, classification_report,roc_auc_score
from scipy.stats import zscore
from sklearn.model_selection import train_test_split
from sklearn import metrics
import os
os.chdir (r"D:\anaconda\ashok\Project\UnsupervisedLearning")
import warnings
warnings.filterwarnings('ignore')
1. Data pre-processing¶
In [2]:
# Loading Dataset - Reading the data as a data frame
df = pd.read_csv('vehicle.csv')
print (df)
In [3]:
# Finding Shape of data
# Inference : There are 846 rows and 19 columns of data
df.shape
Out[3]:
In [4]:
# Finding Type
df.dtypes
Out[4]:
In [5]:
# Viewing first 10 rows of data
df.head(10)
Out[5]:
In [6]:
#Label Encoding the class variable
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
le = LabelEncoder()
columns = df.columns
print(columns)
df['class'] = le.fit_transform(df['class'])
df.shape
Out[6]:
In [7]:
# Viewing first 10 rows of data after label encoding the class variable
df.head(10)
Out[7]:
Performing EDA in below steps:¶
- Finding any Missing Values
- Finding Outliers
- Understanding attributes using descriptive statistics
- Visualizing attribute distribution using univariate and multivariate analysis
- Finding attribute correlation and analysing which attribute is more important
In [8]:
# Checking presence of missing values
df.isna().apply(pd.value_counts)
Out[8]:
In [9]:
df.info()
Inference:¶
- class, hollows_ratio,max.length_rectangularity,max.length_aspect_ratio, compactness has no missing values rest all features are having some kind of missing values </br>
- All attributes are of numerical type
In [10]:
# Finding and treating missing values
from sklearn.impute import SimpleImputer
newdf = df.copy()
X = newdf.iloc[:,0:19] #separting all numercial independent attribute
imputer = SimpleImputer(missing_values=np.nan, strategy='median', verbose=1)
transformed_values = imputer.fit_transform(X)
column = X.columns
print(column)
newdf = pd.DataFrame(transformed_values, columns = column )
newdf.describe().T
Out[10]:
In [11]:
# Checking the results after imputing missing values
print("Original null value count:\n", df.isnull().sum())
print("\n\nCount after imputing null values:\n", newdf.isnull().sum())
The data pre-processing is completed and ready to be fed to an Unsupervised algorithm¶
2. Understanding attributes in the given dataset¶
Univariate Analysis is done to understand the attributes in given dataset. Then, Bivariate analysis is performed to find the relationship between attributes
Below steps are followed:
- Quick descriptive statistics to make some meaningful sense of data </br>
- Plotting univariate distribution </br>
- Finding outliers & skewness in data series </br>
- Treating outliers </br>
- Performing Correlation Analysis using heatmap </br>
- Plotting Bivariate distribution </br>
In [12]:
# Descriptive Statistics Summary of dataframe after imputing null values
newdf.describe().T
Out[12]:
In [13]:
# Finding shape of dataframe after imputing null values
newdf.shape
Out[13]:
Observations based on Descriptive Stats:¶
- compactness has mean and median values almost similar , it signifies that it is normally distribited and has no skewness/outlier
- circularity : it also seems to be normally distribted as mean amd median has similar values
- scatter_ratio feature seems to have some kind of skewness and outlier
In [14]:
#Ploting histogram to check that if data columns are normal or almost normal or not
plt.style.use('seaborn-whitegrid')
newdf.hist(figsize=(15,15), color='blue', edgecolor = 'black')
plt.show()
Observations:¶
- Most of the data attributes seems to be normally distributed
- scaled_variance.1 and skewness_about, skewness_about.1, scatter_ratio seems to be right skewed
In [15]:
# Measuring skewness
newdf.skew(axis = 0, skipna = True)
Out[15]:
In [16]:
#Summary View of all attributes using boxplot
ax = sns.boxplot(data=newdf, orient="h")
In [17]:
# Box Plot to check outliers
plt.figure(figsize= (5,30))
plt.subplot(18,1,1)
sns.boxplot(x= newdf['compactness'], color='blue')
plt.subplot(18,1,2)
sns.boxplot(x= newdf['circularity'], color='blue')
plt.subplot(18,1,3)
sns.boxplot(x= newdf['distance_circularity'], color='blue')
plt.subplot(18,1,4)
sns.boxplot(x= newdf['radius_ratio'], color='blue')
plt.subplot(18,1,5)
sns.boxplot(x= newdf['pr.axis_aspect_ratio'], color='blue')
plt.subplot(18,1,6)
sns.boxplot(x= newdf['max.length_aspect_ratio'], color='blue')
plt.subplot(18,1,7)
sns.boxplot(x= newdf['scatter_ratio'], color='blue')
plt.subplot(18,1,8)
sns.boxplot(x= newdf['elongatedness'], color='blue')
plt.subplot(18,1,9)
sns.boxplot(x= newdf['pr.axis_rectangularity'], color='blue')
plt.subplot(18,1,10)
sns.boxplot(x= newdf['max.length_rectangularity'], color='blue')
plt.subplot(18,1,11)
sns.boxplot(x= newdf['scaled_variance'], color='blue')
plt.subplot(18,1,12)
sns.boxplot(x= newdf['scaled_variance.1'], color='blue')
plt.subplot(18,1,13)
sns.boxplot(x= newdf['scaled_radius_of_gyration'], color='blue')
plt.subplot(18,1,14)
sns.boxplot(x= newdf['scaled_radius_of_gyration.1'], color='blue')
plt.subplot(18,1,15)
sns.boxplot(x= newdf['skewness_about'], color='blue')
plt.subplot(18,1,16)
sns.boxplot(x= newdf['skewness_about.1'], color='blue')
plt.subplot(18,1,17)
sns.boxplot(x= newdf['skewness_about.2'], color='blue')
plt.subplot(18,1,18)
sns.boxplot(x= newdf['hollows_ratio'], color='blue')
plt.show()
Observations on Box Plots:¶
radius_ratio, pr.axis_aspect_ratio, max.length_aspect_ratio, scaled_variance,scaled_variance.1, scaled_radius_of_gyration.1, skewness_about, skewness_about.1 are the attributes with outliers which is visible with all dotted points
In [18]:
# Treating outliers with IQRs
from scipy.stats import iqr
Q1 = newdf.quantile(0.25)
Q3 = newdf.quantile(0.75)
IQR = Q3 - Q1
print(IQR)
In [19]:
# Since number of outliers are less, all the attributes which had outliers have been treated and removed.
cleandf = newdf[~((newdf < (Q1 - 1.5 * IQR)) |(newdf > (Q3 + 1.5 * IQR))).any(axis=1)]
In [20]:
# Plotting the Box Plot to check whether outliers are removed
ax = sns.boxplot(data=cleandf, orient="h")
In [21]:
# Box Plot after removing outliers
plt.figure(figsize= (5,30))
plt.subplot(18,1,1)
sns.boxplot(x= cleandf['compactness'], color='blue')
plt.subplot(18,1,2)
sns.boxplot(x= cleandf['circularity'], color='blue')
plt.subplot(18,1,3)
sns.boxplot(x= cleandf['distance_circularity'], color='blue')
plt.subplot(18,1,4)
sns.boxplot(x= cleandf['radius_ratio'], color='blue')
plt.subplot(18,1,5)
sns.boxplot(x= cleandf['pr.axis_aspect_ratio'], color='blue')
plt.subplot(18,1,6)
sns.boxplot(x= cleandf['max.length_aspect_ratio'], color='blue')
plt.subplot(18,1,7)
sns.boxplot(x= cleandf['scatter_ratio'], color='blue')
plt.subplot(18,1,8)
sns.boxplot(x= cleandf['elongatedness'], color='blue')
plt.subplot(18,1,9)
sns.boxplot(x= cleandf['pr.axis_rectangularity'], color='blue')
plt.subplot(18,1,10)
sns.boxplot(x= cleandf['max.length_rectangularity'], color='blue')
plt.subplot(18,1,11)
sns.boxplot(x= cleandf['scaled_variance'], color='blue')
plt.subplot(18,1,12)
sns.boxplot(x= cleandf['scaled_variance.1'], color='blue')
plt.subplot(18,1,13)
sns.boxplot(x= cleandf['scaled_radius_of_gyration'], color='blue')
plt.subplot(18,1,14)
sns.boxplot(x= cleandf['scaled_radius_of_gyration.1'], color='blue')
plt.subplot(18,1,15)
sns.boxplot(x= cleandf['skewness_about'], color='blue')
plt.subplot(18,1,16)
sns.boxplot(x= cleandf['skewness_about.1'], color='blue')
plt.subplot(18,1,17)
sns.boxplot(x= cleandf['skewness_about.2'], color='blue')
plt.subplot(18,1,18)
sns.boxplot(x= cleandf['hollows_ratio'], color='blue')
plt.show()
In [22]:
'''
Dropping Class column and see the correlation Matrix & Pairplot Before using this dataframe for PCA
as PCA should only be perfromed on independent attribute
'''
cleandf= newdf.drop('class', axis=1)
cleandf.columns
Out[22]:
In [23]:
'''
Using Pearson Correlation Coefficient to see what all attributes are linearly related and also visualize the same
in the seaborns scatter plot
'''
fig, axs = plt.subplots(figsize=(10,10))
sns.heatmap(cleandf.corr(), annot=True, linewidths=.8, ax=axs)
Out[23]:
Inferences from Heat Map¶
- scaled Variance and scaled variance.1 are strongly correlated with correlation coeff: 0.95
- skewness_about.2 and hollows_ratio are strongly correlated with correlation coeff: 0.89
- distance_circularity and radius_ratio are having positive correlation with corr coeff: 0.77
- compactness and circularity are averagely correlated with coeff: 0.68
- radius_ratio and pr.axis_aspect_ratio are averagely correlated with coeff: 0.66
- scaled _variance and scaled_radius_of_gyration are highly correlated with corr coeff: 0.78
- circularity & distance_circularity are highly correlated with corr coeff: 0.79
- pr.axis_rectangularity and max.length_rectangularity are strongly correlated with coeff: 0.81
- scatter_ratio and elongatedness seems to be have strong negative correlation val : -0.97
elongatedness and pr.axis_rectangularity seems to have strong negative correlation, val: 0.95
max_length_aspect_ratio and radius_ratio have average correlation
- pr.axis_aspect_ratio and max_length_aspect_ratio have very little correlation
- scaled_radius_gyration and scaled_radius_gyration.1 have very little correlated
- scaled_radius_gyration.1 and skewness_about are very little correlated
- skewness_about and skewness_about.1 are correlated
- skewness_about.1 and skewness_about.2 are not correlated
In [24]:
# Pair Plot
sns.pairplot(cleandf.iloc[:,1:])
Out[24]:
Observations from Pair Plot¶
- Pair Plot validated same results as observed in heatmap
- scaled variance and scaled variance.1 have very strong positive correlation
- skewness_about_2 and hollow_ratio also have strong positive correation
- scatter_ratio and elongatedness have very strong negative correlation
- elongatedness and pr.axis_rectangularity have strong negative correlation
scaled variance and scaled variance.1 ; elongatedness and pr.axis_rectangularity are strongly correlated. Hence, they need to be dropped or treated carefully before model building
Choosing the right attributes for model building¶
Our objective is to identify the vehicle based on input features. Hence, our main assumption is there is little or no multicollinearity between the features
From above correlation matrix, we could find that there are features with more than 0.9 correlation. so we can decide to get rid of these columns whose correlation is +-0.9 or above.There are 8 such columns:
- max.length_rectangularity
- scaled_radius_of_gyration
- skewness_about.2
- scatter_ratio
- elongatedness
- pr.axis_rectangularity
- scaled_variance
- scaled_variance.1
Another observation is more than 50 % of attributes are highly correlated. The easiest way to deal with Multicolleniarity is to delete or eliminate one of the perfectly correlated features. (Ex: Scaled Variance & Scaled Variance.1 are having strong positive correlation; One can be picked and another one can be dropped.Similarly , elongatedness and pr.axis_rectangularity we can pick one as they have very strong negative correlation). This approach can be used to select the feature we want to carry forward for model analysis.
However, there is a better approach called PCA - Principle Component Analysis which can be used for dimensions reduction
In [25]:
#ploting the count of the target variable
sns.countplot(newdf['class'])
plt.show()
3. Spliting the dataset into training and test set in the ratio of 70:30¶
In [26]:
array = newdf.values
X = array[:,0:18] # select all rows and first 18 columns which are the attributes
Y = array[:,18] # select all rows and the 19th column which is the class variable
test_size = 0.30 # taking 70:30 training and test set
# Scaling of independent variable data using standard scaler
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_std = sc.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_std, Y, test_size=test_size, random_state=1) # To set the random state
type(X_train)
Out[26]:
4. Training a Support Vector Machine¶
In [27]:
svc = SVC() #instantiate the object
#fit the model on orighinal raw data
svc.fit(X_train,y_train)
Out[27]:
In [28]:
#predict the y value
Orig_y_predict = svc.predict(X_test)
In [29]:
print("Model Score On Original Data - training dataset ",svc.score(X_train, y_train))
print("Model Score On Original Data - test dataset ",svc.score(X_test, y_test))
print("Accuracy Score On Original Data",accuracy_score(y_test, Orig_y_predict))
In [30]:
print(metrics.confusion_matrix(y_test, Orig_y_predict))
Observation¶
Accuracy on test data set using support vector machine on original data is 0.952755905511811
5. Performing K-fold cross validation and getting cross validation score for the model¶
In [31]:
from sklearn.model_selection import KFold
scores = []
best_svc = SVC(kernel='rbf')
cv = KFold (n_splits=10, random_state=1, shuffle=False)
for train_index, test_index in cv.split(X_std):
X_train, X_test, y_train, y_test = X_std[train_index], X_std[test_index], Y[train_index], Y[test_index]
best_svc.fit(X_train, y_train)
scores.append(best_svc.score(X_test, y_test))
print(scores)
Observation¶
The cross validation scores on test data of original dataset (raw data) after performing K-fold cross validation for K=10 is as shown above. </br>
The minimum accuracy score is 0.9411764705882353 and maximum accuracy score is 0.9882352941176471
6.Using PCA , extracting Principal Components that capture about 95% of the variance in the data¶
Principal Components Analysis is done using below steps¶
- Split our data into train and test data set (already performed in section-3 above)
- normalize the training set using standard scalar (already performed in section-3 above)
- Create the covariance matrix
- Calculate the eigenvectors and their eigenvalues
- Sort the eigenvectors according to their eigenvalues in descending order
- Choose the first K eigenvectors (K - required dimension)
- Build new dataset with reduced dimensionalityIn [32]:
# Creating covariance matrix
cov_matrix = np.cov(X_std.T)
print("cov_matrix shape:",cov_matrix.shape)
print("Covariance_matrix",cov_matrix)
In [33]:
# Calculating Eigen Vectors & Eigen Values: Using numpy linear algebra function
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
print('Eigen Vectors \n%s', eigenvectors)
print('\n Eigen Values \n%s', eigenvalues)
In [34]:
# Sorting eigenvalues in descending order
# Make a set of (eigenvalue, eigenvector) pairs:
eig_pairs = [(eigenvalues[index], eigenvectors[:,index]) for index in range(len(eigenvalues))]
# Sort the (eigenvalue, eigenvector) pairs from highest to lowest with respect to eigenvalue
eig_pairs.sort()
eig_pairs.reverse()
print(eig_pairs)
# Extract the descending ordered eigenvalues and eigenvectors
eigvalues_sorted = [eig_pairs[index][0] for index in range(len(eigenvalues))]
eigvectors_sorted = [eig_pairs[index][1] for index in range(len(eigenvalues))]
# print out eigenvalues
print('Eigenvalues in descending order: \n%s' %eigvalues_sorted)
In [35]:
tot = sum(eigenvalues)
var_explained = [(i / tot) for i in sorted(eigenvalues, reverse=True)] # an array of variance explained by each
# eigen vector... there will be 18 entries as there are 18 eigen vectors)
cum_var_exp = np.cumsum(var_explained) # an array of cumulative variance. There will be 18 entries with 18 th entry
# cumulative reaching almost 100%
In [36]:
# Plotting The Explained Variance and Principal Components
plt.bar(range(1,19), var_explained, alpha=0.5, align='center', label='individual explained variance')
plt.step(range(1,19),cum_var_exp, where= 'mid', label='cumulative explained variance')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.legend(loc = 'best')
plt.show()
Observation¶
- From above plot, it is clear that 8 dimensions are able to explain 95% variance of data
- Hence, First 8 principal components can be used going forward to calculate the reduced dimensions
Dimensionality Reduction¶
- 8 dimensions are used to explain 95% of variation in original dataIn [37]:
# Building new data set with reduced dimensionality
P_reduce = np.array(eigvectors_sorted[0:8]) # Reducing from 8 to 4 dimension space
X_std_8D = np.dot(X_std,P_reduce.T) # projecting original data into principal component dimensions
reduced_pca = pd.DataFrame(X_std_8D) # converting array to dataframe for pairplot
reduced_pca
Out[37]:
In [38]:
# Pairplot Of Reduced Dimensions After PCA
sns.pairplot(reduced_pca, diag_kind='kde')
Out[38]:
Observation from above Pair Plot¶
After dimensionality reduction using PCA, the attributes have become independent with no correlation among themselves. As most of them have cloud of data points with no linear relationship
7. Fitting SVC Model and measuring accuracy scores using Principal Components¶
In [39]:
# Building Support Vector Classifier Model with the 8 new reduced variables constructed using PCA
# Spliting the dataset into training and test set in the ratio of 70:30
pca_X_train,pca_X_test,pca_y_train,pca_y_test = train_test_split(reduced_pca,Y,test_size=0.30,random_state=1)
In [40]:
#Fitting the model on PCA data with new dimensions
svc1 = SVC()
svc1.fit(pca_X_train,pca_y_train)
Out[40]:
In [41]:
#predicting the y value
pca_y_predict = svc1.predict(pca_X_test)
In [42]:
print("Model Score On Reduced PCA Dimension ",svc1.score(pca_X_test, pca_y_test))
print("Accuracy score after PCA(On 8 dimensions)",accuracy_score(pca_y_test,pca_y_predict))
In [43]:
#Printing Confusion matrix
print(metrics.confusion_matrix(pca_y_test,pca_y_predict))
In [44]:
# Performing K-fold cross validation using principal components
from sklearn.model_selection import KFold
pca_scores = []
pca_svc = SVC(kernel='rbf')
pca_cv = KFold (n_splits=10, random_state=1, shuffle=False)
for pcatrain_index, pcatest_index in pca_cv.split(X_std_8D):
X_train, X_test, y_train, y_test = X_std_8D[pcatrain_index], X_std_8D[pcatest_index], Y[pcatrain_index], Y[pcatest_index]
pca_svc.fit(X_train, y_train)
pca_scores.append(pca_svc.score(X_test, y_test))
print(pca_scores)
Observations¶
- Accuracy score after PCA(On 8 dimensions) is: 0.937007874015748
- The cross validation scores on Principal Components data after performing K-fold cross validation for K=10 : The minimum accuracy score is 0.9285714285714286 and maximum accuracy score is 0.9764705882352941
8.Comparing the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components¶
SVM model Scores - Original Data (18 dimensions):¶
- Accuracy Score On Original Data 0.952755905511811
- The cross validation scores on original data after performing K-fold cross validation for K=10 is:
- Minimum accuracy score is 0.9411764705882353
- Maximum accuracy score is 0.9882352941176471
- Confusion Matrix: [[ 57 1 1] [ 1 129 3] [ 6 2 54]]
SVM model Scores - Data using Principal Components (8 dimensions):¶
- Accuracy score after PCA(On 8 dimensions) is: 0.937007874015748
- The cross validation scores on Principal Components data after performing K-fold cross validation for K=10 :
- Minimum accuracy score is 0.9285714285714286
- Maximum accuracy score is 0.9764705882352941
- Confusion Matrix: [[ 57 1 1] [ 2 126 5] [ 5 2 55]]
Conclusion:¶
Using support vector classifier:
- On raw dataset with 18 dimensions, the accuracy score without performing PCA is 95.27 %
- On PCA components with only 8 dimensions(reduced dimensions) our model scored 93.70 %.
Considering that original dataframe had 18 dimensions, after PCA, dimensions got reduced to 8, SVM model has fared well in terms of accuracy score
Confusion Matrix Analysis on Original Data:¶
[[ 57 1 1] [ 1 129 3] [ 6 2 54]]
SVM model on original data set has correctly classified :
- 57 van out of 59 actuals vans and has errored only in 2 cases
- 129 cars out of 133 actual cars
- 54 buses out of 62 actual buses
Wrong classification is minimal
Confusion Matrix Analysis on Reduced Dimenions After PCA :¶
[[ 57 1 1] [ 2 126 5] [ 5 2 55]]
SVM model on original data set has correctly classified :
- 57 van out of 59 actuals vans
- 126 cars out of 133 actual cars
- 55 buses out of 62 actual buses
By using only 8 out of 18 dimensions, wrong classification is minimal and results are comparable with the one we got for original data
In [ ]: